Validation of surgical care quality indicators in the Brazilian Unified Health System

ABSTRACT OBJECTIVE To validate a set of indicators for monitoring the quality of surgical procedures in the Brazilian Unified Health System (SUS). METHODS Validation study developed in 5 stages: 1) literature review; 2) prioritization of indicators; 3) content validation of indicators by RAND/UCLA consensus method; 4) pilot study for reliability analysis; and 5) development of instruction for tabulation of outcome indicators for monitoring via official information systems. RESULTS From the literature review, 217 indicators of surgical quality were identified. The excluded indicators were: those based on scientific evidence lower than 1A, similar, specific, which corresponded to sentinel events; and those that did not apply to the SUS context. Twenty-six indicators with a high level of scientific evidence were submitted to expert consensus. Twenty-two indicators were validated, of which 14 process indicators and 8 outcome indicators with content validation index ≥80%. Of the validated process indicators, 6 were considered substantially reliable (Kappa coefficient between 0.6 and 0.8; p < 0.05) and 2 had almost perfect reliability (Kappa coefficient > 0.8, p < 0.05), when the inter-rater agreement was analyzed. One could measure and establish tabulation mechanism for TabWin for 7 outcome indicators. CONCLUSION The study contributes to the development of a set of potentially effective surgical indicators for monitoring the quality of care and patient safety in SUS hospital services.


INTRODUCTION
The Brazilian Unified Health System (SUS) performs about five million surgeries annually, mostly elective surgical procedures 1 . Such therapeutic resource has been increasingly regarded as an essential component of public health, its role growing in importance with the increase in life expectancy 2 . However, little is known about the quality and safety of surgeries performed in SUS. This is a crucial gap since, despite their benefits, surgeries also present risks to the patient and costs to the health system. Data shows 312.9 million surgeries were performed in 2012 worldwide 2 , an increase of about 36.8% since the launch of the Second Global Challenge for Patient Safety, Safe Surgeries Saves Lives 3 .
The Ministry of Health, health sector regulatory agencies and non-governmental bodies have supported initiatives to improve the quality and safety of surgeries through actions related to the elaboration of public policies 4 , technical standards and regulations for inspection and monitoring purposes. However, there is still a lack of a standardized set of indicators for monitoring surgeries in SUS. Such monitoring is important since it enables quality improvement and provides learning to teams, in addition to enabling the development of regulatory capacity, being essential for a good clinical performance 5,6 .
In the last decade, indicators have been developed to guide initiatives for improvement of quality in perioperative care 7,8 and to stimulate positive changes towards achieving quality at a reasonable cost 9 . These indicators are used as direct measures of the quality and safety of the care provided; however, they are still insufficient. Therefore, we are dealing with a scenario in which the existing indicators are not standardized and consolidated, nor periodically measured by the care network, leading to a void of important information and lack of comparability between existing information, negatively affecting the planning and quality management of care in the SUS.
Thus, this study aims to identify and validate a minimum set of process and outcome indicators that can be used to monitor the quality of surgical procedures in SUS.

METHODS
This study is part of the QualiCir Project, an intervention project aimed at improving the quality and safety of surgical procedures in the state of Rio Grande do Norte (RN), and is developed in partnership with the QualiSaúde Research Group of the Federal University of Rio Grande do Norte and the RN Public Health Secretariat. This is a methodological study on the validation of perioperative quality indicators applicable to elective surgical procedures performed in SUS. The study was developed in 5 stages: 1) literature review; 2) selection of indicators for consensus; 3) content validation of indicators; 4) pilot study for reliability analysis; and 5) development of instructions for tabulation of outcome indicators.

Stage 1 -Literature review:
A search was performed in PubMed and Google Scholar databases, looking for articles of current systematic reviews (< 5 years of publication). As search strategy, the keywords "quality indicators" and "surgical procedures" were included. Searches were also carried out on official State websites and documents, pursuing indicators developed by national organizations regarded as reference in the promotion of patient care and safety, so to obtain a list of potential indicators to be used to measure surgical quality in the Brazilian context. Indicators were selected from regulatory agencies in the health sector 10,11 , Patient Safety Indicators (ISEP-Brazil Project) 12 , Health System Performance Assessment Project (PROADESS) 13 , and the Collaborating Center for Quality and Patient Safety (PROQUALIS) 14 .

Stage 2 -Selection of indicators for consensus:
Based on the indicators found in the previous step, those that had the following criteria were selected: a) aspects related to the entire surgical process; b) high scientific evidence (1A); c) able to evaluate the quality of surgical care in any hospital of the national health system; d) can be used to implement improvement measures based on their results. Indicators that were similar amongst themselves, sentinels, not applied to the SUS context, that evaluate a specific surgical procedure or patient group, with contradictory evidence, and indicators that present measurement difficulties (many components of measurements, unclear) were excluded.

Stage 3 -Content validation of the indicators:
Validation was performed using the RAND/UCLA method 15 , which associates aspects of the Delphi and Nominal Group methods 14 and combines the observation of the available scientific evidence with the collective judgment of experts. The validation of indicators is done through a consensus opinion derived from a group, with aggregated individual opinions, which is an established approach for the development of health indicators 5 . The group of specialists consisted of eight surgeons and two nurses. Nine members of this group of specialists worked in public institutions in four different Brazilian states, and one was a Spanish surgeon who coordinated a similar study in his country.
Two rounds of consensus were established: the first occurred by completing the electronic questionnaire sent by email and the second was developed by web conferencing.
A questionnaire was developed using the Google Forms platform, based on similar studies 12,14,16 , containing five closed questions for each indicator, using a Likert-type scale for responses. The following criteria were used for the evaluation and selection of indicators: 1) Is the indicator clearly relevant?; 2) Does the indicator measure the quality of care or safety in surgical care?; 3) Can the indicator be modified with improvement interventions implemented by the hospital?; 4) Are the data for the indicator measurement possible to collect?; and 5) Is the wording of the indicator clear, with correct terminology and leaving no doubts?
Indicators that obtained a content validation index (CVI) greater than 80% 17 in the five proposed items would be considered valid for the measurement of surgical quality. Indicators that did not reach this value in the first round were taken to the second round.
As a subsidy for the two rounds, an indicator form was developed containing the following information: title, measure, justification, indicator type, data source, numerator and denominator description, clarifications/definition of terms, limitations/exceptions, and bibliographic references. Collection was carried out by two independent evaluators, with previous experience in collecting data from medical records, in a cross-sectional manner, in samples of 30 medical records each, referring to elective surgeries occurred in 2020, selected systematically 18,19 .
The adequacy of indicators by sample type was established by consulting experts. Most of the process indicators were evaluated in sample A1, with the exception of the indicators "Timely removal of surgical nasogastric tubes" and "Early removal of bladder catheter", which were evaluated in sample 2.
Stage 5 -Identification of tabulation mechanism for result indicators so that they can be monitored via official information systems -The validated result indicators were analyzed for their possibility of monitoring through the use of data from official information systems, from the identification of tabulation mechanism for TabWin/DataSus with the Hospital Information System of SUS (SIH-SUS -Sistema de Informações Hospitalares do SUS) database.
The research was carried out under the approval of the Research Ethics Committee of the Federal University of Rio Grande do Norte (CEP-HUOL, CAAE: 39976920.6.0000.5292), following the ethical precepts in research with human beings, according to resolution CNS/MS 466/12.

RESULTS
217 quality or safety indicators related to surgical procedures, totaling 183 process indicators and 34 outcome indicators were found. The choice to use the content of systematic reviews as the main reference for the literature search was made to avoid the repetition of a recent study with similar objectives.
Of the 183 process indicators, 138 were excluded by the criterion of low scientific evidence (< 1A) ( Figure 1). Although the level of evidence of the indicator "Use of safe Source: Prepared by the authors.   Continue surgery checklist" is not high, the researchers decided to keep this indicator in the study due to its regulation in Brazilian health services. Twelve indicators were excluded because they were considered similar, two because they were not applied to the sus, five because they were indicators applied to a very specific public or procedure, eight did not allow the development of improvement cycles and two were based on contradictory scientific evidence.
As for the outcome indicators, 10 indicators were excluded because they were considered similar, eight were very specific, two did not allow the development of improvement cycles and four were related to sentinel events. At the end of this trial, 16 process indicators and 10 outcome indicators were submitted to content validation with the group of experts. The selection flow of indicators can be seen in Figure 1.
In the first round, which was attended by 100% of the invited experts, validation questionnaires were sent by email and 26 indicators were presented to the group. In this round, the 13 indicators that received CVI greater than 80% were considered valid for measuring surgical quality within the SUS. The other 13 indicators, due to achieving CVI equal to or less than 80% in any of the evaluated criteria, were submitted to the second round of consensus. This step occurred through web conferencing and was attended by 80% of the invited experts. Discussions on indicators with CVI ≤ 80% took place at the time and, subsequently, a new evaluation was carried out, as can be seen in Chart 1.
At the end of the second round, four indicators received CVI ≤ 80% and were not considered valid: the indicator "Preoperative use of oral carbohydrates", which presented CVI of 75% in the criterion related to the writing of the indicator; the indicator "Improved recovery" had CVI of 75% in the criteria related to the availability of data for measurement and clarity in the writing; the indicators "Post-surgical stroke" and "Unscheduled admission to an intensive care unit" obtained CVI of 75% in the criteria related to the availability of data and the possibility of modifying the indicator through improvement interventions. Thus, 22 indicators were considered valid for the measurement of quality in surgeries, of which 14 were process and 8 were outcome indicators. The data source, numerator and denominator of these indicators are described in Chart 2.
The qualification sheets of the validated indicators were reformulated according to suggestions of the experts, with the addition and reformulation of terms and concepts.
To analyze the reliability of the indicators, whose data source are the medical records, a retrospective pilot study was carried out at the Due to the HRMC qualification profile, it was not possible to collect the indicators "Postoperative discharge with postoperative evaluation, prophylaxis of venous thromboembolism and postoperative rehabilitation", and "Record of pressure and time during controlled ischemia in surgery". The search for another institution of the state hospital network that was qualified to perform orthopedic surgeries to evaluate these indicators was considered; however, this was not possible given the low number of orthopedic elective surgeries performed in 2020 due to the covid-19 pandemic, in addition to the lack of pneumatic tourniquet in the hospital institutions that make up the state network.
As for the reliability analysis, six indicators showed substantial reliability and two almost perfect reliability 20 , as can be seen in Table 1. One could not measure the reliability for the process indicators "Control of normothermia in the perioperative period", "Screening of postoperative delirium", "Prophylaxis of adequate perioperative venous thromboembolism" and "Use of safe surgery checklist", since the percentage of compliance for these indicators was 0% for both evaluators.
For outcome indicators, whose data source is SIH-SUS, it was observed that seven of the eight validated indicators can be monitored from the TabWin/DATASUS tabulator. Data are publicly accessible and available at https://datasus.saude.gov.br/ transferencia-de-arquivos/.
It was not possible to perform tabulation for the indicator "Post-surgical readmission". As this is a system that analyzes hospital production, it does not link hospitalizations to an individual user record, i.e., through the system one cannot identify how many times a single user was admitted to the hospital, nor is it possible to ascertain whether one admission would be related to the previous one. Table 1. Analysis of the reliability of surgical quality indicators according Landis and Koch (1977) parameters and percentage of compliance achieved.  An instruction was prepared to tabulate the result indicators for the TabWin/DATASUS application for teams that will collect data and monitor it. All results obtained with the other indicators can be seen in Table 2.

DISCUSSION
This study contributed to the development of a set of 22 indicators with a high level of evidence, which underwent a rigorous content validation process to enable the monitoring of the quality of surgical care within the SUS. These indicators can guide the management of institutions and of the hospital network as a whole, identifying weaknesses that must be addressed, aiming at providing safe care to the population. This is, therefore, an initial set of highly relevant indicators for monitoring and improving the quality of surgical care within the scope of SUS RN, with the possibility of being used by any other health service.
From the process indicators, one may evaluate all the steps and activities performed in the implementation of a treatment or care episode 8 . Thus, continuously monitoring these indicators enables one to identify weaknesses in the provision of care. According to Donabedian, process indicators are the only direct measure of quality, as the structure may not be used and outcomes may be due to factors other than good care 21 .
Monitoring of the outcome indicators "Post-surgical mortality", "Post-surgical readmission" and "Average length of stay with and without death" through the information system enables the measurement of the quality of an isolated health service, as well as benchmarking. That is, it enables the comparison of health services from the state hospital network and also at the national level, which strengthens information systems 22 .
The post-surgical mortality indicator is among the indicators proposed by the Lancet Commission 23 to assess surgical care. A similar study 16 developed for the Spanish health system also pointed out the indicators: "Post-surgical readmission", "Prophylaxis of venous thromboembolism", "Adequate antibiotic prophylaxis" and "Surgical site infection" as valid indicators to assess surgical quality; however, these indicators are directed only to surgeries of the digestive tract.
Benchmarking has been used to seek opportunities for improvement and make comparisons of similar organizations 16,24 . It has been listed as a strategy by the World Health Organization (WHO) in the Global Action Plan for Patient Safety 2021-2030 22 , and the development of "good" indicators is a success factor for benchmarking actions 25 .
In addition, 11 indicators could be measured with the available data sources (medical records and data from the official information system), of which 8 process indicators were evaluated in medical records and 3 outcome indicators were measured with SIH-SUS data, exploring the feasibility of using this system to evaluate the quality of surgical care. For the indicators "Screening for postoperative delirium", "Use of safe surgery checklist" and "Prophylaxis of adequate perioperative venous thromboembolism", one should institutionalize protocols related to these indicators, which signals an opportunity for improvement for the hospital where the pilot was developed.
The inter-rater reliability, tested by Kappa statistics for eight process indicators, found values that characterize a substantial and almost perfect degree of reliability, which reinforces the solidity of these indicators. The Kappa test is considered adequate to evaluate the reliability of inter-rater categorical and nominal variables, and is frequently used to evaluate the reliability in this type of study 20 .
For the Surgical Site Infection (SSI) indicator, whose data sources may be medical records or system data, it was not possible to analyze the reliability, since the event was not observed in the medical records selected to compose the sample. Most SSIs occur, on average, four to six days after the procedure, and the average length of stay for the procedures included in the study was 1.5 days. Studies indicate that, in procedures in which the postoperative length of stay is short, SSI data, obtained only from hospitalized patients, do not ref lect the actual occurrence of infection 26 . There was a four-fold increase in SSI when post-discharge surveillance was performed 27 , which leads one to the finding that patient's medical record does not prove to be the best source of data for monitoring this indicator for the vast majority of procedures performed by the SUS.
For the outcome indicators "Complications related to anesthesia", "Postoperative sepsis", "Pulmonary edema or deep vein thrombosis", measurement via the information system was not possible. The results were null, possibly due to underreporting of secondary events in the Hospital Admission Authorization (AIH) forms. A study on the reliability of AIH data in the country identified a high degree of underreporting of secondary diagnosis 28 . The underreporting of secondary diagnosis in surgical admissions impacts the accuracy of measures calculated for these indicators, which is an opportunity for improvement for the health information system.
The Minimum Health Care Data Set (CMD), conceived in 2015, is a strategy assumed by managers of the three SUS management spheres to reduce fragmentation of information systems, and would replace the main health care information systems in the country. However, despite having been officially instituted by resolution of the Tripartite Intermanagerial Commission 29 , its implementation has not yet been completed. The CMD implementation would enable the use of administrative, clinical-administrative, and clinical data through a single document, in addition to enabling more specific analyzes, since it would relate the information to the identification of users through integration with the base of the National Health Card system. Despite the efforts and studies carried out in the field of patient safety, the ability to reduce risk, avoid harm, and improve health care safety is still hampered by the absence of high-quality information systems 22 .
The review of existing literature and consensus methods are increasingly used and recommended by the scientific community for this type of study 16,30 . The use of the RAND/UCLA method to establish consensus, through the use of remote communication resources (internet), allowed to bring together qualified specialists from various regions of the country. The interest of experts in the studied area, associated with the observed consensus indexes, gave credibility to the results, as can be seen in other studies 14,31 .
As limitations of this study, we can highlight the performance of the pilot study in a single hospital, whose care profile did not include surgical procedures of the musculoskeletal system, as well as the conduct of the pilot study in a pandemic period, which decreased the sample universe, due to the cancellation of elective surgeries throughout the hospital network. Other limitations, which may be the subject of further studies, are the nonassessment of structural indicators and the non-performance of the feasibility analysis for the collection of indicators.

CONCLUSION
This study contributed to the development of a set of quality indicators in the surgical sphere, which translates as an effective mechanism for measuring the performance and quality of care offered by the hospital service network of RN and Brazil. There are 22 indicators that were considered valid, with 8 process indicators considered reliable and seven result indicators, in which parameters were identified for tabulation using the official information systems. This set of indicators enables the documentation of quality of care, enables comparisons and benchmarking between health units, promotes the